The Correlation of Machine Translation Evaluation Metrics with Human Judgement on Persian Language

Machine Translation Evaluation Metrics (MTEMs) are the central core of Machine Translation (MT) engines as they are developed based on frequent evaluation. Although MTEMs are widespread today, their validity and quality for many languages is still under question. The aim of this research study was to examine the validity and assess the quality of MTEMs from Lexical Similarity set on machine translated Persian texts. This study focused on answering three main questions, which included the extent that Automatic Machine Translation Evaluation Metrics is valid on evaluating translated Persian texts; the probable significant correlation between human evaluation and automatic evaluation metrics in evaluating English to Persian translations; and the best predictor of human judgment. For this purpose, a dataset containing 200 English sentences and their four reference human translations, was translated using four different statistical Machine translation systems. The results of these systems were evaluated by seven automatic MTEMs and three human evaluators. Then the correlations of metrics and human evaluators were calculated using both Spearman and Kendall correlation coefficients. The result of the study confirmed the relatively high correlation of MTEMs with human evaluation on Persian language where GTM proved to be more efficient compared with other metrics.